Image Processing Apparatus Capable of Highly Precise Edge Extraction 

[0001] This application is based on Japanese Patent Application No. 
2003-80314 filed with Japan Patent Office on March 24, 2003, the entire 
content of which is hereby incorporated by reference. 
BACKGROUND OF THE INVENTION 
Field of the Invention 

[0002] The present invention relates to an image processing apparatus, . 
an image processing program product and an image pick-up apparatus. 
More specifically, the present invention relates to an image processing 
apparatus, an image processing program product and an image pick-up 
apparatus that are capable of extracting edge portions with high precision. 
Description of the Related Art 

[0003] Conventionally, a technique has been proposed in which each 
pixel of an image prepared by extracting edge components from an image 
input through a camera or the like (hereinafter referred to as an edge 
image) is compared in magnitude with a threshold value set in advance, to 
form an image having edge portions and non-edge portions distinguished 
from each other (hereinafter referred to as an edge binarized image). 
When the threshold value is set large in such a technique, it becomes 
difficult to extract a portion that is desirable to be extracted as the edge 
portion, if that portion is a thin line with small gradation difference. On 
the contrary, when the threshold value is set small, it becomes possible to 
extract that portion of the input image which has small gradation 
difference. However, it becomes more likely that edge components that 
exist around the original edge component that has been successfully 
extracted are undesirably extracted as edge portions as well, resulting in 
an edge portion that is too thick. Further, there is a higher possibility that 
not only a thin fine but also noise components would undesirably be 
extracted. Thus, it has been very difficult to determine a right threshold 
value that can simultaneously solve both of the above described problems. 
[0004] Japanese Laid-Open Patent Publication No. 2000-13607 
discloses an image processing method in which an image prepared by 



smoothing an input image (hereinafter referred to as a smoothed image) is 
subtracted from the input image, an edge component is extracted based on 
the result of subtraction, and the edge component is binarized by using a 
prescribed threshold value, whereby an edge portion and a non-edge 
portion are distinguished from each other. 

[0005] The image processing method disclosed in Japanese Laid-Open 
Patent Publication No. 2000-13607 will be specifically described. 
[0006] Fig. 9 represents an image A-l having large gradation difference, 
an image B-l having a small gradation difference, and an image C-l having 
moderate gradation with the gradation difference changing like a wave. 
According to the image processing method disclosed in Japanese Laid-Open 
Patent Publication No. 2000-13607, first, the input images are smoothed 
using an average filter. Edge smoothed images A-2, B-2 and C-2 obtained 
by averaging the input images of Fig. 9 by taking 7 pixels by 7 pixels are 
shown in Fig. 10. 

[0007] Thereafter, according to the image processing method disclosed 
in Japanese Laid-Open Patent Publication No. 2000-13607, difference 
between the input image (or the image for outline extraction) and the 
smoothed image is calculated. Results of subtraction A-3, B~3 and C-3 of 
smoothed images A-2, B-2 and C-2 shown in Fig. 10 from the input images 
A-l, B-l and C-l shown in Fig. 9 are shown in Fig. 11. 
[0008] Japanese Laid-Open Patent Publication No. 2002-175534 
discloses a method in which an edge image is formed from an input image, 
and using an average value and standard deviation of the edge component 
of the edge image, an edge binarized image is formed. 
[0009] The method disclosed in Japanese Laid-Open Patent Publication 
No. 2000-13607 has a problem that when there is a portion having wave- 
like gradation difference where the gradation changes moderately, the peak 
portion of the wave that should not be extracted as an edge is undesirably 
extracted as an edge portion. More specifically, in the method disclosed in 
Japanese Laid-Open Patent Publication No. 2000-13607, the portion © is 
extracted as the edge portion based on the result of subtraction B-3 shown 
in Fig. 11, and therefore, when the threshold value is set to 8 as an 



example, portions © and ® of input image A- 1 would be extracted as the 
edge portion based on the result of subtraction A-3. Thus, the extracted 
edge portion becomes rather thick. When the edge portion is detected 
thick, it becomes sometimes difficult to detect accurate position of an object. 
Further, in the result of subtraction C-3, values of portions ©to © are 
higher than the threshold value 8, and therefore, these portions in input 
image C-l are undesirably extracted as the edge portion, though these 
portions actually have the wave-like gradation difference with moderately 
changing gradation. 

[0010] The method disclosed in Japanese Laid-Open Patent Publication 
No. 2002-175534 also has a problem that it is difficult to extract, as an edge 
portion, a portion having small gradation difference such as a thin like 
existing in an input image of which portions mostly have large gradation 
difference. 

SUMMARY OF THE INVENTION 

[00 1 1] Therefore, an object of the present invention is to provide an 
image processing apparatus, an image processing program and an image 
pick-up apparatus that are capable of extracting edge portions with high 
precision. 

[00 12] The above described object of the present invention is attained 
by an image processing apparatus including the following elements. 
Specifically, the image processing apparatus includes an edge image 
forming unit. extracting an edge from an input image and forming an edge 
image, an edge smoothed image forming unit smoothing the edge image to 
from an edge smoothed image, a difference calculating unit calculating 
difference between the edge image and the edge smoothed image, and a 
binarizing unit binarizing the edge image based on the difference. 
[0013] According to another aspect, the present invention provides an 
image processing program product that makes a computer execute an 
image processing, including an edge smoothed image forming step of 
smoothing an edge image formed based on an input image and forming an 
edge smoothed image, a difference calculating step of calculating difference 
between the edge image and the edge smoothed image, and a binarizing 



step of binarizing the edge image based on the difference. 
[00 14] According to a still further aspect, the present invention 
provides an image pick-up apparatus including an image pick-up unit 
picking-up an object and capturing an object image, an edge image forming 
unit forming an edge image by extracting an edge from the picked-up image, 
an edge smoothed image forming unit forming an edge smoothed image by 
smoothing the edge image, a difference calculating unit calculating 
difference between the edge image and the edge smoothed image, and a 
binarizing unit binarizing the edge image based on the difference. 
[0015] The foregoing and other objects, features, aspects and 
advantages of the present invention will become more apparent from the 
following detailed description of the present invention when taken in 
conjunction with the accompanying drawings. 
BRIEF DESCRIPTION OF THE DRAWINGS 

[0016] Fig. 1 shows a specific exemplary configuration of an image 

processing system in accordance with an embodiment. 

[00 17] Fig. 2 is a flow chart representing the process executed by the 

image processing system in accordance with the embodiment. 

[0018] Fig. 3 represents specific examples of edge images. 

[0019] Fig. 4 represents specific examples of edge smoothed images 

averaged by taking 7 pixels by 7 pixels of the input image. 

[0020] Fig. 5 represents specific examples of results of subtraction of 

the edge smoothed images from the edge images. 

[002 1] Figs. 6 and 7 illustrate the principle of edge portion extraction. 

[0022] Fig. 8 shows a specific exemplary configuration of a camera 2, 

when image processing is done by the camera 2. 

[0023] Fig. 9 represents specific examples of input images. 

[0024] Fig. 10 represents specific examples of edge smoothed images 

averaged by taking 7 pixels by 7 pixels of the input image. 

[0025] Fig. 11 represents specific examples of results of subtraction of 

the edge smoothed images from the input images. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0026] Embodiments of the present invention will be described with 



reference to the figures. In the following description, same or 
corresponding elements are denoted by the same reference characters. 
Names and functions of these portions are also the same. Therefore, 
detailed description thereof will not be repeated. 

[0027] Referring to Fig. 1, the image processing system in accordance 
with the present invention includes a camera 2 inputting images to an 
image processing apparatus such as a personal computer (hereinafter 
referred to as a PC), and PC 1 processing the images obtained from camera 
2. The image processing apparatus of the present invention is not limited 
to the personal computer as specifically represented in the present 
embodiment, and the apparatus may be implemented as a semiconductor 
chip, or as an image processing board. 

[0028] Further referring to Fig. 1, PC 1 as the image processing 
apparatus is controlled by a CPU (Central Processing Unit) 101, and 
processes an image input from camera 2 through a camera I/F (interface) 
107 (which is also referred to as an image capturing unit). The program 
executed by CPU 101 is stored in a HDD (Hard Disk Drive) 102 or a ROM 
(Read Only Memory) 103 as a storing unit. Alternatively, the program 
executed by CPU 101 is read by a reading unit 108 from a storage medium 
109 such as a CD-ROM (Compact Disc-ROM). A RAM (Random Access 
Memory) 104 serves as a temporary work space when the program is 
executed by CPU 101. Further, RAM 104 also serves as a buffer, that is, a 
temporary storage area, for the following processes. A user inputs 
information and instructions through an input unit 105 including a 
keyboard and a mouse. Images received from camera 2 and results of 
processing thereof are displayed on a display unit 106. The configuration 
shown in Fig. 1 is a general configuration of a personal computer, and the 
configuration of PC 1 is not limited to that shown in Fig. 1. 
[0029] Camera 2 may be any general apparatus, such as a video 
recorder, which has means for capturing an image and for inputting the 
same to PC 1. 

[0030] In such an image processing system, the following image 
processing is executed, an edge image is formed from the image captured by 
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camera 2, and the edge image is binarized. Fig. 2 is a flow chart 
representing the process executed by the image processing system in 
accordance with the present embodiment, realized by the program read by 
CPU 101 of PC 1 from HDD 102 or ROM 103 or the program obtained by 
reading unit 108 from storage medium 109, and executed by RAM 104. 
[0031] Referring to Fig. 2, first, CPU 101 of PC 1 receives an input of 
an object image OI (x, y) from camera 2 through camera I/F 107 (Si). Here, 
the input object image OI (x, y) input from camera 2 is either a color image 
or a gray scale image. 

[0032] Thereafter, CPU 101 executes the program to transform the 
input object image OI (x, y) to a gray scale image GI (x, y) (S2). In step S2, 
when the input object image OI (x, y) is a color image, the calculation 
represented by Equation (1) below is performed on each pixel of the input 
object image OI (x, y), to transform the input object image OI (x, y) to the 
gray scale image GI (x, y): 

GI(x,y) = 0.299 xROI(x,y) 
+ 0.587 xGOI(x,y) 

+ 0.114xBOI(x,y) ---(l) 
where (x, y) represents coordinates indicating the pixel position in the 
image, ROI (x, y) represents an R plane of the input object image OI (x, y) 
that is a color image, BOI (x, y) represents a B plane of the input object 
image OI (x, y) that is a color image, and GOI (x, y) represents a G plane of 
the input object image OI (x, y) that is a color image. 

[0033] When the input object image OI (x, y) input in step Si is already 
a gray scale image GI (x, y), the process of step S2 is omitted, and the 
process proceeds to the next step. 

[0034] Next, CPU 101 extracts an edge component from the gray scale 
image GI (x, y) transformed in step S2, and forms an edge image EI (x, y) 
(S3). As to the method of forming an edge image EI (x, y) in step S3, 
various conventional methods may by utilized, and not specifically limited 
in the present invention. By way of example, a method may be used in 
which the calculation represented by Equation (2) below is performed on 
each pixel of the gray scale image GI (x, y): 



EI (x, y) = 4(Gl (x, y ) - GI (x, y - + ( GI (x, y ) - GI (x - 1, y )f 
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[0035] The forming method mentioned above is generally well-known, 
by which the difference value between adjacent pixels is extracted as an 
edge component. 

[0036] Alternatively, a method using SOBEL operator represented by 
Equation (3) or a method using PREWITT operator represented by 
Equation (4) may also be utilized, to similarly form the edge image EI (x, y). 
Further, similar effects can be attained by forming the edge image EI (x, y) 
using a method of extracting edge components other than the methods 
described below. 
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[0037] Here, EIx (x, y) and Ely (x, y) represent an edge image with 
edge component along the X direction extracted and an edge image with 
edge component along the Y direction extracted, respectively. 
[0038] Thereafter, CPU 101 performs a smoothing process on the edge 
image EI (x, y) formed in step S3, to form an edge smoothed image EHI (x, 
y) (S4). In step S4, the edge smoothed image EHI (x, y) is formed by using 
an average filter. Specifically, a method represented by Equation (5) 
below using an average filter of 7 pixels X 7 pixels is used. 



EHI(x,y) = 
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[0039] The method of forming the edge smoothed image EHI (x, y) in 
step S4 is not limited to the method represented by Equation (5) above, and 
similar processing is possible using an average filter of different size. A 
larger filter size is advantageous in that the edge image EI (x, y) can be 
made smoother, while it is disadvantageous in that amount of calculation 
increases. Through trials, the inventors have found that the filter size of 5 
pixels X 5 pixels to 1 1 pixels X 11 pixels is preferable, and 7 pixels X 7 
pixels given in Equation (5) is most preferable. 

[0040] Thereafter, CPU 101 calculates the difference Diff between edge 
image EI (x, y) formed in step S3 and edge smoothed image EHI (x, y) 
formed in step S4, in accordance with Equation (6) below. 

Diff = EI(x, y) - EHI(x, y) • • • (6) 

[0041] Then, CPU 101 binarizes edge image EI (x, y) in accordance 
with the value of difference Diff calculated in step S5, to form an edge 
binarized image EB (x, y) (S6). Here, a pixel of which value of difference 



Diff is larger than a preset threshold value TH is regarded as an edge 
portion, to form the edge binarized image EB (x, y). 
[0042] In the manner described above, image processing by PC 1 is 
completed, and an edge binarized image can be obtained from the image 
captured by camera 2. 

[0043] The value of difference Diff calculated in step S5 tends to be 
relatively large at a portion having a small gradation difference but to be 
extracted as an edge portion, and the value tends to be very small where 
the gradation difference changes moderately like a wave. Such tendency 
will be described with reference to specific examples of image processing 
above. Though the input image is considered as one-dimensional for 
convenience of description in the following, it would be readily understood 
that the same applies similarly to two-dimensional images. 
[0044] Assume that PC 1 receives the images shown in Fig. 9 described 
above, i.e. the image A-l having large gradation difference, the image B-l 
having small gradation difference, and the image C-l having gradation 
difference moderately changing like a wave, as inputs from camera 2. 
CPU 101 of PC 1 calculates the difference between adjacent pixels of 
respective input images A-l, B-l and C-l in step S3 described above to 
extract edge components, and forms the edge images A-4, B-4 and C-4 
shown in Fig. 3, respectively. 

[0045] Thereafter, CPU 101 of PC 1 subtracts respective edge smoothed 
images A-5, B-5 and C-5 from edge images A-4, B-4 and C-4, to obtain 
results of subtraction A-6, B-6 and C-6, as shown in Fig. 5. Specifically, 
the following calculations are executed to obtain the results A-6, B-6 and C- 
6: 

(A-6) = (A-4) - (A-5) 

(B-6) = (B-4) - (B-5) 

(C-6) = (C-4) - (C-5). 
[0046] Here, if the threshold value is set to 18, for example, to extract 
the portion (7) of edge image B-4 as an edge portion, portions (5) to CD 
would be extracted in edge image A-4, resulting in a thick edge portion. 
[0047] When the threshold value is set to 15, for example, to extract the 



portion (7) of input image B-l as an edge portion based on the result of 
subtraction B-6, only the portion ® in input image A-l is extracted as an 
edge portion based on the result of subtraction A-6. Namely, edge portions 
can be extracted successfully from both the portion having large gradation 
difference and the portion having small gradation difference. Further, it 
is noted that every pixel value is much lower than the threshold value TH = 
15 in the result of subtraction C-6, and therefore, an edge portion is not 
extracted from the portion having the gradation difference changed 
moderately like a wave. 

[0048] This can be explained with reference to Figs. 6 and 7. 
Specifically, when the input image has gradation difference, the pixel 
values of the edge image change considerably as shown in Fig. 6. 
Therefore, the portion of the edge image having high pixel values, that is, 
the edge portion, comes to have large difference from the edge smoothed 
image. Accordingly, by setting a prescribed threshold value in 
consideration of the difference between the edge image and the edge 
smoothed image, the edge portion can be extracted. 

[0049] When the gradation of the input image changes moderately, the 
change in pixel values of the edge image is small, as can be seen from Fig. 7. 
Therefore, the difference between the edge image and the edge smoothed 
image comes to be very small. Accordingly, by setting a prescribed 
threshold value in consideration of the difference between the edge image 
and the edge smoothed image, it becomes possible not to extract the peak 
portion of the wave of the moderately changing gradation as the edge 
portion. 

[0050] As described above, when the method in which the edge is 
binarized based on the difference between the edge image and the edge 
smoothed image is used, it becomes possible to extract the portion having a 
large gradation difference without making the edge portion thick. Further, 
the peak portion of the wave of moderately changing gradation is not 
extracted as an edge portion, and hence, edge detection with higher 
precision becomes possible, 

[0051] In the embodiment above, it is described that a color image is 
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transformed to a gray scale image and thereafter edge component is 
extracted, in steps S2 and S3. Similar effects can be attained when the 
process of step S3 is performed to extract the edge component in each of the 
RGB planes of the color image. Further, where it is possible to obtain 
information close to that of a gray scale image by using only one of the RGB 
planes of the color image, the process of step S3 may be performed only for 
one color plane. 

[0052] Though it is described in the embodiment above that image 
processing is performed by PC 1 on the image captured by camera 2, image 
processing may be performed by camera 2, if camera 2 has such a 
configuration as shown in Fig. 8. Referring to Fig. 8, when camera 2 is 
controlled by a CPU 201 and performs processing of the image picked-up by 
image pick-up unit 204, the above described image processing may be 
performed by control unit 201 executing the program. In this case, the 
program to be executed by CPU 201 of camera 2 is stored in a ROM 202 as 
a storage unit. RAM 203 serves as a temporary storage area when the 
program is executed by CPU 201. The user inputs information and 
instructions through an operating unit 205 including buttons or the like. 
The image picked-up by image pick-up unit 204, result of processing thereof 
and so on are output from an output unit 206. Output unit 206 may be a 
display unit implemented by a liquid crystal panel, or it may be an I/F for 
transmitting data to other apparatuses. The configuration shown in Fig. 8 
represents a general configuration of a digital camera and the like, and the 
configuration of camera 2 here is not limited to the one shown in Fig. 8. 
[0053] Further, the method of image processing performed by PC 1 or 
camera 2 as the image processing apparatus described above may be 
provided as a program. Such a program may be recorded on a computer 
readable recording medium such as a flexible disk, CD-ROM, ROM or 
memory card to be used with a computer, and provided as a program 
product. Alternatively, the program may be provided recorded on a 
recording medium such as a hard disk mounted inside the computer. 
Further, the program may be downloaded though a network. 
[0054] The provided program product is installed in a program storing 
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unit such as a hard disk and executed. The program product encompasses 
the program itself and the recording medium on which the program is 
recorded. 

[0055] Although the present invention has been described and 
illustrated in detail, it is clearly understood that the same is by way of 
illustration and example only and is not to be taken by way of limitation, 
the spirit and scope of the present invention being limited only by the terms 
of the appended claims. 
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